Bandit Linear Optimization for Sequential Decision Making and Extensive-Form Games

Tree-form sequential decision making (TFSDM) extends classical one-shot by modeling tree-form interactions between an agent and a potentially adversarial environment. It captures the online decision-making problems that each player faces in extensive-form game, as well Markov processes partially-observable where conditions on observed history. Over past decade, there has been considerable effort into designing optimization methods for TFSDM. Virtually all of work full-feedback setting, access to counterfactuals, is, information what would have happened had chosen different action at any node. Little is known about bandit assumption reversed (no counterfactual available), despite this latter setting being understood almost 20 years making. In paper, we give first algorithm linear problem TFSDM offers both (i) linear-time iterations (in size tree) (ii) O(sqrt(T)) cumulative regret expectation compared fixed strategy, times T. This made possible new results derive, which may independent uses well: 1) geometry dilated entropy regularizer, 2) autocorrelation matrix natural sampling scheme sequence-form strategies, 3) construction unbiased estimator losses 4) refined analysis mirror descent when using regularizer.